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SERIAL NO. : 



GROUP ART UNIT: 



FILED: CONCURRENTLY 



EXAMINER: 



MOTION ESTIMATION 



Honorable Commissioner of Patents and Trademarks 
Washington, D.C. 20231 



Sir: 



PRELIMINARY AMENDMENT 
Prior to calculating the filing fee and examination 
please amend the above-identified application as follows: 

IN THE SPECIFICATION 
Page 1, before line 1, insert as a centered heading 
— BACKGROUND OF THE INVENTION — ; 
after the heading, insert at the left margin 
— Field Of The Invention — ; 
between lines 4 and 6, insert at the left margin 

— Description Of The Related Art — ; 
lines 11, 13, 20, 21 and 28, after "i.e." insert — 
(comma) ; 

line 14, after "Therefore" insert — , — (comma); 
line 16, after "e.g." insert — , — (comma); 
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line 22, change "candidate the" to — candidate, the — ; 
line 25, change "WO-A-97/46, 022" to — International Patent 

Application No. WO-A-97/46, 022, corresponding to 

U.S. Patent Application Serial No. 08/863,700, 

filed May 27, 1997,—; 
Page 2, line 1, after "e.g." insert — , — (comma); 

line 3, change "EP-A-0, 652, 678" to — European Patent 

Application EP-A-0, 652 , 678 , corresponding to U.S. 

Patent 5,473,379, — ; 
between lines 7 and 9, insert as a centered heading 

— SUMMARY OF THE INVENTION — ; 
lines 12 and 13, delete in their entirety, and insert 

— apparatus . — ; 
line 16, change "vectors is" to — vectors, is — ; 
line 18, after "vector" insert (comma); 
between lines 22 and 24, insert as a centered heading 

— BRIEF DESCRIPTION OF THE DRAWING — ; 
between lines 25 and 27, insert as a centered heading 

— DESCRIPTION OF THE PREFERRED EMBODIMENTS—; 
line 27, after "as" insert — a--; 
Page 3, line 2, after "vector" insert — , — (comma); 

after "First" insert — , — (comma); 
line 12, after "count" insert — , — (comma); 
line 13, after "blocks" insert — , — (comma); 
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line 14, after "counts" insert — , — {comma); 



Page 



line 11, after "i.e." insert — , — {comma); 



?age 



line 23, after "i.e." insert — (comma); 



Page 



6, 



lines 13, 2S, 31 and 32, after "e.g." insert — , — 



( comma ) ; 

Page 1, line 8, delete "In"; 

line 9, delete in its entirety; 
line 10, delete "the claim.". 

IN THE ABSTRACT 

Page 10, before line 1, delete in its entirety, and insert as a 
centered heading 

— ABSTRACT OF THE DISCLOSURE--. 



IN THE CLAIMS 



Please amend the claims as follows: 
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1. (J\mended) A motion vector estimation method/ comprising the 
steps [of] : . 

carrying out a block-based motion vector estimation 
process [ (BME) ] that involves comparing a plurality of candidate 
5 vectors to determine block-based motion vectors; 

determining at least a most frequently occurring block- 
based motion vector [ (MFMV) ] ; 

carrying out a global motion vector estimation process 
[(GME)] using at least the most frequently occurring block-based 
10 motion vector [ (MFMV) ] to obtain a global motion vector [ (GMV) ] ; 
and 

applying the global motion vector [ (GMV) ] as a candidate 
vector to the block-based motion vector estimation process [ (BME) ] . 

2. (Amended) [A] The method as claimed in claim 1, wherein the 
determining step includesj_ 

making a selection among block-based motion vectors having 
a corresponding motion error below a given motion error threshold. 

3. (Amended) [A] The method as claimed in claim 1, wherein the 
determining step includes^ 
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making a selection among block-based motion vectors 
estimated for blocks having a difference between maximum and 
5 minimum pixel values above a given activity threshold. 



4. (Amended) [A] The method as claimed in claim 1, wherein 
both the most frequently occurring block-based motion vector 

[ (MFMV) ] and a second-most frequently occurring block-based motion 
vector [ (SMFMV) ] are determined and used in the global motion 
5 vector estimation process [ (GME) ] . 

5. (Amended) [A] The method as claimed in claim 1, wherein 
said global motion vector estimation process [ (GME) ] includes the 
steps [of] ; 

comparing, on a block basis, a plurality of candidate 
5 vectors_^ including the most frequently occurring block-based motion 
vector_f_ [ (MFMV) ] to obtain best vectors determined per block; and 

outputting a most-f requently occurring best vector 
determined per block as the global motion vector [ (GMV) ] . 

6. (Amended) A motion vector estimation device, comprising: 

block-based motion vector estimation means [ (BME) ] for 
determining block-based motion vectors based on a comparison of a 
plurality of candidate vectors; 
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means for determining at least a most frequently occurring 
block-based motion vector [ (MFMV, SMFMV) ] ; 

■ means [ (GME) ] for carrying out a global motion vector 
estimation process using at least the most frequently occurring 
block-based motion vector [ (MFMV, SMFMV) ] to obtain a global motion 
vector; and 

means for applying the global motion vector [ (GMV) ] as a 
candidate vector to the block-based motion vector estimation means 
[ ( BME) ] . 

7. (Amended) A motion-compensated picture signal processing 
apparatus, comprising: 

a motion vector estimation device as claimed in claim 6 
for generating motion vectors; and 

a motion-compensated processor [ (MCP) ] for processing a 
picture signal in dependence on the motion vectors. 

8. (Amended) A picture display apparatus, comprising: 

a motion-compensated picture signal processing apparatus 
as claimed in claim 7 to obtain a processed picture signal; and 
a display device for displaying the processed picture 

signal . 
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REMARKS 

The specification has been amended in various places to 
correct typographical and graininatical errors. The specification has 
also been amended to add section headings. 

The claims have been amended to more clearly define the 
invention as disclosed in the written description. In particular, 
the claims have been amended for clarity. 

When the Examiner takes this case up for examination, it 
is respectfully requested that this Preliminary Amendment be taken 
into consideration. 



Respectfully submitted, 




Edward W. Goodman, Reg. 28,613 
Attorney 

Tel.: 914-333-9611 
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Motion estimation. 



The invention relates to a method and device for motion estimation, a motion- 
compensated picture signal processing device comprising such a motion estimation device, 
and a picture display apparatus comprising such a motion-compensated picture signal 
processing device. 

5 

A prior art motion estimation technique, called "3-D Recursive Search", has 
been described by Gerard de Haan and Robert Jan Schutten, "Real-time 2-3 pull-down 
elimination applying motion estimation/compensation in a programmable device", IEEE 
Transactions on Consumer Electronics, Vol. 44, No. 3, August 1988, pp. 930-938. 3-D 

10 Recursive Search falls in the class of pixel- or block-recursive motion estimators. The 

algorithm is based in the following assumptions: motion does not change much in time, i.e. 
from frame to frame. The algorithm maintains a motion field and tries to update this field only 
when necessary. The motion field is usually similar for a relatively large region, i.e. for an 
object. Therefore the motion vectors in the neighborhood of a location are good candidates for 

15 the motion in that location. Video consists of a sequence of frames. Each frame is divided into 
blocks, e.g. of 16x16 pixels. A motion vector is associated with each block. The motion vector 
should hold the displacement between the block in the current frame compared to the previous 
frame. Suppose that we want to update the motion vector of block (x,y) in the current frame. 
3-D Recursive Search uses only a limited number of candidate vectors, say five, for the 

20 estimation, viz. some vectors from the previous frame, i.e. temporal vectors, some vectors 
from the current frame, i.e. spatial vectors, and an update of a spatial vector. For each 
candidate the motion estimation error is calculated. The candidate with the lowest motion 
estimation error is chosen as the best motion vector for that block. The algorithm uses the 
normal raster scan order to go through the blocks. 

25 WO-A-97/46,022 discloses a method of estimating motion vectors, in which 

motion parameters are determined for a given field of a video signal, and motion vectors for a 
subsequent field of the video signal are determined in dependence upon at least one 
predetermined motion vector (i.e. a motion vector already estimated for a spatio-temporally 
neighboring block) and at least one additional motion vector derived from the motion 

30 parameters. The motion parameters for the given field may be derived from motion vectors 
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determined for the given field, e.g. by applying a two-dimensional histogram operation on the 
motion vectors determined for the given field. 

pP-A-0,652,678 discloses a method and apparatus for improving a block-based 
motion compensation in digital video coding. The location of the search window within a 
5 reference frame is defined using the global motion of the frame. In one embodiment, the 
global motion vector is generated utilizing the motion vector occurring with the most 
repetition within a plurality of previously stored motion vectors. 

It is, inter alia, an object of the invention to provide an improved motion 
10 estimation technique. To this end, the invention provides a motion estimation method and 
device, a motion-compensated picture signal processing apparatus, and a picture display 
apparatus as defined in the independent claims. Advantageous embodiments are defined in the 
dependent claims. 

In a motion vector estimation method in accordance with a primary aspect of 
1 5 the present invention, a block-based motion vector estimation process that involves comparing 
a plurality of candidate vectors is carried out to determine block-based motion vectors, at least 
a most frequently occurring block-based motion vector is determined, a global motion vector 
estimation process using at least the most frequently occurring block-based motion vector is 
carried out to obtain a global motion vector, and the global motion vector is applied as a 
20 candidate vector to the block-based motion vector estimation process. 

These and other aspects of the invention will be apparent from and elucidated 
with reference to the embodiments described hereinafter. 

The drawing shows a functional block diagram of an embodiment of a motion- 
25 compensated picture display apparatus in accordance with the present invention. 

When using 3-D Recursive Search for global motion estimation, we have as 
main task: how to obtain one motion vector from all the motion vectors? Our approach is 
based on the following steps: 
30 1 . The global motion vector is defined as the most used motion vector in 

appropriate blocks. We also use the second-most used motion vector. A block is appropriate if 

the motion estimation error is small enough and the block contains enough detail. 

2. Make the motion field smooth by introducing a global motion vector candidate. 
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In fact, we use two motion estimators, a normal motion estimator that is using 
the global motion vector and a global motion estimator. First we will describe how we extract 
a global motioti from the motion field. We will describe the properties of each motion 
estimator in the next sections. Finally, it is described how both motion fields are used to build 
5 the global motion estimator. 

Why do we need two motion vectors? We use the normal motion estimator to 
track the changes. From the associated motion field we caimot obtain the correct global 
motion, but only candidate global motion vectors. We use a global motion estimator, still 
using the 3-D Recursive Search concept, for selecting the best global motion vector. We 
10 cannot solely use this global motion estimator because it is not capable of tracking changes. 

The global motion is extracted from the motion field in two steps: count for all 
"appropriate" blocks the number of times that a motion vector is used, and obtain from these 
counts the most and second-most used motion vectors. 

1 5 A block is "appropriate" if the motion estimation error is small enough (average 

SAD smaller than 30), and the block contains enough activity (activity larger than 50) where 
the activity per block is defined as: activity = max(ij)(y(i,j)) - min(ij)(y(i,j)). We remove the 
blocks with a low activity, because the motion estimation is not reliable for blocks without 
detail. Currently, we use the Sum-of- Absolute-Difference measure (SAD) for the displacement 

20 error. Let y[i,j] and yprev [ij] denote the pixel values of the current frame and previous frame, 
respectively. The Sum-of- Absolute-Difference measure is calculated by: 

SAD = Y\y{iJ)-y^^^Xi^MV^J + MV^) [ 

where (MVx,MVy) is the candidate motion vector and the summation is over the block. The 
SAD is set to "0" if it is smaller than a threshold to remove the influence of fixed pattern noise. 
25 We also use the second-most used motion vector to improve the robustness of 

the algorithm. We found that sometimes the algorithm will favor the zero-motion vector, 
whereas there is some camera parming. Supplying the global motion estimator with both 
vectors solves this situation. 

30 The 3-D Recursive Search estimator uses the following 6 candidates: 

1 . The most-used global motion vector (which is also used as best global motion 
vector). 

2. The spatial vector of block (x- l,y- 1) (upper left). 
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3. The spatial vector of block (x+ l,y- 1) (upper right). 

4. The temporal vector of the current block. 

5 . . The temporal vector of block (x,y+ 1 ) (lower) . 

6. An update of the spatial vector of block (x- 1, y- 1) if x is even and of block 
5 (x+ 1 ,y- 1 ) if X is odd. 

The update is obtained as follows. The update vector is the sum of the spatial 
vector and a delta vector. The delta vector (dx, dy) is read from a list of 16 possible delta 
vectors. The list of sixteen possible delta vectors is given in the next table. 

10 



j 


0 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


dx 


0 


0 


0 


0 


4 


-4 


12 


-12 


0 


0 


1 


-1 


0 


0 


2 


-2 


dy 


4 


-4 


8 


-8 


0 


0 


0 


0 


1 


-1 


0 


0 


2 


-2 


0 


0 



The table shows that the maximum update per vector is 12 pixels horizontally and 8 pixels 
vertically. 



How do we select a delta vector? We simply use the next delta vector in the list 
for the next block and start with delta vector 0. Suppose that element j is used for block (x,y), 
1 5 then we use element j+1 for the next block (i.e. block (x+1 ,y)) and we use delta element "0" if 
j+1 equals sixteen. 

Each candidate is checked to see whether the resulting address is valid, i.e. 
points to an area within the frame. If not, the vector is clipped to the nearest valid motion 
vector. 

20 

The next table shows the penalties for each vector type. Penalties are added to 
block matching errors (SAD) in order to favor certain candidate motion vectors over other 
candidate motion vectors in order to smoothen the motion field. 



Vector type 


Penalty (per block) 


Global motion vector 


3 


Spatial vector 


0 


Temporal vector current block 


0 


Temporal vector lower block 


8 


Update vector 


32 



25 



The global motion estimator uses the following four candidates: 



PHN 17.569 



5 13.01.2000 

1 . Most-used motion vector obtained by the block-based 3-D Recursive Search 
estimator. 

2. , .Second-most used motion vector obtained by the block-based 3-D Recursive 
Search estimator. 

5 3. Cyclically varying updates of the motion vector mentioned at 1 (see above). 

4. Cyclically varying updates of the motion vector mentioned at 2 (see above). 

On a block basis, the global estimator determines which of the four candidates 
is the best one. From these best candidates determined on a block basis, the most-frequently 
10 occurring one is retained. 



The penalties for each type are: 



Vector type 


Penalty (per block) 


Motion vector 


Oor 1 


Update of motion vector 


32 



The penalty for the global motion candidate is 1 if the motion vector is zero and 
0 otherwise. This may be simplified to "0" only without losing accuracy. 

15 

The full motion estimator in accordance with the present invention uses the 
following steps per frame: 

1 . Get the best global motion vector from the global motion estimation GME. 

2. Use this vector for the 6-candidate 3-D Recursive Search motion estimation 
20 BME. 

3 . Extract the most used and second-most used global motion vectors from the 
resulting motion field. 

4. Use these motion vectors in the global motion estimation GME, i.e. in the four- 
candidate motion estimation. 

25 5. The global motion is extracted from the resulting motion field and used in step 

1. 

To reduce the CPU load of the algorithm, the number of motion estimation 
blocks is reduced by sub-sampling. Since we only require one motion vector per frame, the 
30 global motion vector, we do not need to calculate a motion vector for each block, so that the 
number of used blocks can be sub-sampled. We currently use a sub-sampling factor of two 
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horizontally and two vertically. Note that we may be able to use a factor of four for the global 
motion estimation, if necessary. The sub-sampling factor is limited for the following reasons: 
A too high sub-sampling factor reduces the probability that there are "appropriate candidates" 
(blocks with a small motion estimation error and a sufficiently high activity). Moreover, using 
5 too few blocks will reduce the smoothness of the motion field. In addition, it is possible to 
apply sub-sampling within a block to reduce the number of pixels. 

Using processor-specific features, such as MMX, also helps in speeding up the 
computation. Also, the time spend in the SAD calculation can in principle be reduced by using 
cross correlation. 

10 To improve the global motion estimation algorithm, the following measures are 

possible. Retain not only the most used and second-most used global motion vectors, but also 
less firequently used motion vectors. Use only the central part of the current fi-ame for motion 
estimation, e.g. a quarter of the frame. If there is some rotation (with the middle of the frame 
as center of rotation), the blocks in the outer area of the frame will contain more displacement 

1 5 than the central part. Note that this latter measure will also reduce the computational load. 

The drawing shows a functional block diagram of an embodiment of a motion- 
compensated picture display apparatus in accordance with the present invention. A picture 
signal is applied to a block-based motion vector estimator BME and to a global motion- vector 

20 estimator GME that operate as set out above. The block-based motion vector estimator BME 
applies a most frequently used motion vector MFMV and a second-most frequently used 
motion vector SMFMV to the global motion-vector estimator GME. The global motion-vector 
estimator GME applies a global motion vector GMV as a candidate vector to the block-based 
motion vector estimator BME. The picture signal is also applied to a motion-compensated 

25 processor MCP for carrying out, e.g. a motion-compensated interpolation (say, a 1 00 Hz 
conversion) or a motion-compensated stitching of images obtained by a scanner or video 
camera. The motion-compensated processor is controlled by either block-based motion vectors 
supplied by the block-based motion vector estimator BME or global motion vectors supplied 
by the global motion estimator GME. A switch S symbolically indicates this choice. In 

30 practice, depending on the application, there is no switch S and the appropriate type of motion 
vectors is used. Global vectors will e.g. be used for stitching scanned images, while block- 
based vectors will e.g. be used for 100 Hz conversion. The output of the motion-compensated 
processor MCP is applied to a display device DD. In other applications of the invention, such 
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as in a scanner, the output of the motion-compensated processor MCP will be printed on 
paper. 

It should be noted that the above-mentioned embodiments illustrate rather than 
5 limit the invention, and that those skilled in the art will be able to design many alternative 
embodiments without departing from the scope of the appended claims. Where in the above- 
described examples only a most-used and a second-most used vector are used, it is an obvious 
generalization clearly falling within the scope of the claims to use the N most-used vectors. In 
the claims, any reference signs placed between parentheses shall not be construed as limiting 

10 the claim. The word "comprising" does not exclude the presence of elements or steps other 
than those listed in a claim. The word "a" or "an" preceding an element does not exclude the 
presence of a plurality of such elements. The invention can be implemented by means of 
hardware comprising several distinct elements, and by means of a suitably programmed 
computer. In the device claim enumerating several means, several of these means can be 

15 embodied by one and the same item of hardware. The mere fact that certain measures are 
recited in mutually different dependent claims does not indicate that a combination of these 
measures cannot be used to advantage. 
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CLAIMS: 



1 . A motion vector estimation method, comprising the steps of: 
carrying out a block-based motion vector estimation process (BME) that 

involves comparing a plurality of candidate vectors to determine block-based motion vectors; 

determining at least a most frequently occurring block-based motion vector 

5 (MFMV); 

carrying out a global motion vector estimation process (GME) using at least the 
most frequently occurring block-based motion vector (MFMV) to obtain a global motion 
vector (GMV); and 

applying the global motion vector (GMV) as a candidate vector to the block- 
1 0 based motion vector estimation process (BME). 

2. A method as claimed in claim 1, wherein the determining step includes making 
a selection among block-based motion vectors having a corresponding motion error below a 
given motion error threshold. 

15 

3. A method as claimed in claim 1, wherein the determining step includes making 
a selection among block-based motion vectors estimated for blocks having a difference 
between maximum and minimum pixel values above a given activity threshold. 

20 4. A method as claimed in claim 1 , wherein both the most frequently occurring 

block-based motion vector (MFMV) and a second-most frequently occurring block-based 
motion vector (SMFMV) are determined and used in the global motion vector estimation 
process (GME). 
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5. A method as claimed in claim 1 , wherein said global motion vector estimation 
process (GME) includes the steps of: 

comparing, on a block basis, a plurality of candidate vectors including the most 
frequently occurring block-based motion vector (MFMV) to obtain best vectors determined 
5 per block; 

outputting a most-frequently occurring best vector determined per block as the 
global motion vector (GMV). 

6. A motion vector estimation device, comprising: 

1 0 block-based motion vector estimation means (BME) for determining block- 

based motion vectors based on a comparison of a plurality of candidate vectors; 

means for determining at least a most frequently occurring block-based motion 
vector (MFMV, SMFMV); 

means (GME) for carrying out a global motion vector estimation process using 
15 at least the most frequently occurring block-based motion vector (MFMV, SMFMV) to obtain 
a global motion vector; and 

means for applying the global motion vector (GMV) as a candidate vector to the 
block-based motion vector estimation means (BME). 



20 7. A motion-compensated picture signal processing apparatus, comprising: 

a motion vector estimation device as claimed in claim 6 for generating motion 

vectors; and 

a motion-compensated processor (MCP) for processing a picture signal in 
dependence on the motion vectors. 

25 

8. A picture display apparatus, comprising: 

a motion-compensated picture signal processing apparatus as claimed in claim 7 
to obtain a processed picture signal; and 

a display device for displaying the processed picture signal. 
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In a motion vector estimation method, a block-based motion vector estimation 
process (BME) that involves comparing a plurality of candidate vectors is carried out to 
determine block-based motion vectors, at least a most frequently occurring block-based 
motion vector (MFMV) is determined, a global motion vector estimation process (GME) using 
5 at least the most frequently occurring block-based motion vector (MFMV) is carried out to 
obtain a global motion vector (GMV), and the global motion vector (GMV) is applied as a 
candidate vector to the block-based motion vector estimation process (BME). 
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;=^d belief are believed to be true; and further that these statements were made with the knowledge that willful false statements and the 
^ like so made are punishable by fine or imprisonment, or both, under Section 1 001 of Title 1 8 of the United States Code and that such 
%illful false statements may jeopardize the validity of the application or any patent issued thereon. 



CSOWER OF ATTORNEY: As a named inventor, 1 hereby appoint the following attorney(s) and/or agent(s) to prosecute this application 
and transact all business in the Patent and Trademark Office connected therewith, (list name and registration number) 



Algy Tamoshunas, Reg. No. 27,677 




Jack E. Haken, Reg. No. 26,902 




SEND CORRESPONDENCE TO: Corporate Patent Counsel; 


DIRECT TELEPHONE CALLS TO: 


U.S. Philips Corporation; 580 white Plains Road; 


(name and telephone No.) 


Tarrytown, NY 10591 


(914) 332-0222 



Dated: Inventor's Signature: 

June 28, 2000 ^^^^ 


Full Name of in 
Inventor 


Last Name 
BEUKER 


First Name 
Rob 


Middle Name 
A. 


Residence & 
Citizenship 


City 

Eindhoven 


State of Foreign Country 
The Netherlands 


Country of Citizenship 
The Netherlands 


Post Office Address 


Street 

Prof, l-lolstlaan 6 


City 

5656 AA Eindhoven 


State of Country 1 Zip Code 
The Netherlands I 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re Application of Atty. Docket 

ROB A. BEUKER PHN 17,569 

Serial No. ^ Group Art Unit: 

Filed: CONCURRENTLY Examiner: 
Title: MOTION ESTIMATION 

Honorable Commissioner of Patents and Trademarks 
Washington, D.C. 20231 



APPOINTMENT OF ASSOCIATES 

Sir: 

The undersigned Attorney of Record hereby revokes all 
prior appointments (if any) of Associate Attorney (s) or Agent (s) 
in the above-captioned case and appoints: 

Edward W. Goodman (Registration No. 28, 613) and 

c/o U,S. PHILIPS CORPORATION, Intellectual Property Department, 
580 White Plains Road, Tarrytown, New York 10591, 
his Associate Attorney (s) /Agent (s) with all the usual powers to 
prosecute the above- identified application and any division or 
continuation thereof, to make alterations and amendments therein, 
and to transact all business in the Patent and Trademark Office 
connected therewith. 



ALL CORRESPONDENCE CONCERNING THIS APPLICATION AND THE 
LETTERS PATENT WHEN GRANTED SHOULD BE ADDRESSED TO THE 
UNDERSIGNED 
ATTORNEY OF RECORD. 

Respect^<f3?y , 



Dated at Tarrytown, New York 
this 20™ day of July, 2000. 




Tamoshunas , Reg . 27,577 
Attorney of Record 
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